PhD Thesis Robust Reconstruction and Efficient Localization for Mobile Augmented Reality

This thesis introduces robust reconstruction and efficient localization methods for mobile Augmented Reality (AR). Robust Structure from Motion(SfM) methods are necessary to create sparse reconstructions of specific areas where localization and therefore AR should be possible. The resulting reconstructions serve as large, general tracking targets. Efficient localization methods are required to solve the localization problem on computationally limited mobile devices. Apart from specific algorithmic improvements, this thesis demonstrates that image-based localization and SfM are highly interconnected problems. The 3D data that is needed for pose computation is exactly what SfM methods compute. Furthermore, image-based pose estimation algorithms can minimize a reprojection error and not an object-space error. This reprojection error is exactly what matters for superimposing information, which is the topic of AR. The user notices localization deviations as pixel offsets, the absolute position error is usually irrelevant. We present a robust incremental hierarchical SfM approach that allows to create large data sets of specific areas for localization applications. Image matches are the input of basically all SfM methods. Because image matching itself is a hard task we increase the matching robustness further by using the additional sequential information of image sequences when possible and by screening the epipolar graph with novel cycle constraints in graphs of pair-wise visual relations. Highly efficient localization methods are presented that use visibility information to partition the SfM point cloud results. This addresses two limitations of current mobile devices for image-based localization: (i) The amount of main memory that is necessary for search data structures and (ii) computational requirements. We show that localization quality is highly correlated with the field of view. Since panoramic images can be created online on mobile devices, the user can contribute to the localization performance by increasing the field of view. The robustness and performance of the presented reconstruction and localization methods is finally demonstrated and evaluated in a large-scale outdoor localization experiment that uses a combination of all presented techniques.